DDD Data Load

Prerequisites

The following are the prerequisite conditions for DDD Data Load:

  1. To configure Reltio tenant details, see Configure Connection.

  2. To configure notification channels such as email id or MS teams channel or slack channel, see Adding Alert Notification. Configure at least one notification channel and Subscribe to the task group pipeline.

  3. Keep the DDD data files and code file ready. Code files should be in generic code import supported layout. See Codes sheet in this DDD DID.

  4. Keep MDM_Extract pipeline executed once and make sure RDM data tables are created.

  5. Create a folder with name as DDD/input under s3 default root folder.

  6. Task group MDM_Load_Refresh_Reltio_Views_Customer_Master should be imported and successfully executed atleast once per OA/IDP MDM release.

DDD Data Load Overview

DDD data is loaded to Landing and Staging tables using SCD Type1 Full Refresh.

DDD Data from Staging History table is transformed as intermediate views. The data from these views are loaded to canonical tables using generic procedure.

DDD RDM data codes are loaded till Reltio RDM tenant.

Note:   

To avoid the mapping issues while extracting the RDM codes from two different tenants in the same IDP environment, each RDM_SOURCEMAPPINGS table name is configured with a suffix specifying the Reltio connection name as IDPSchema.MV_@ReltioConnectionName_RDM_SOURCEMAPPINGS. The RDM_SRCMAPPINGS view references from Staging RDM table for which, this process manages the transaction and sync it from Land to Stage.

Import Pipeline Template

To Import Pipeline Template

  1. Connect to IDP default s3 bucket and go to the folder bucket_name>/templates/product.

  2. Download the MDM_Load_Reltio_DDD<version>.json file template to local windows folder.

    Note:   

    If there are multiple files prefixed MDM_Generic_RDM_Code_Import_* with different versions, download the latest version.

  3. Open the downloaded template json file with any text editor to find and replace the below placeholders with appropriate values.

    Placeholder Replaceable String
    <RELTIO_CONN_NAME> Reltio Connection name configured in Entity Collection. For example, RELTIO_MDM

    Note:   

    The search strings given above are case-sensitive, do not enclose with any character, replace as it is.

  4. Login to IDP platform with valid credentials.

  5. Go to Data Pipeline app, click Task Group from Template and import the template using updated template file.

    Note:   

    Process creates the pipeline task group after the successful import. This has to be used for execution.

  6. Note that this task group execution has a capability to create MIDAS Product Reference data which is used in data bridging application. By default this reference data creation is disabled. If user wants to create reference data then, set the value of parameter Create_DDD_Reference_Table to true in this task group. Also, if enabled this feature then, do not use/execute the other task group Bridge_DDD_Product_Reference_Load which also creates DDD HCO reference data.

Operational Steps

The following are the operational steps:

  1. Keep the data files ready as per the layout defined in the DDD DID.

  2. Login to IDP console and open the MDM_Load_Reltio_DDD task group.

  3. Using Winscp or S3 browser, login to default S3 bucket path mentioned and go to the root folder. Place the data files inside the <s3 root folder>/DDD/input folder.

  4. Once files are placed, click Run in IDP platform to trigger the pipeline task.

  5. Click Show Execution Logs which is next to Run button for verifying the task status.

  6. Monitor jobs:

    1. Both Data pipeline Jobs or API call jobs used to post data to Reltio can be monitored through spring cloud data flow (SCDF).

    2. To monitor, Login to IDP portal, click Data Pipeline in Landing Page.

    3. Identify the Task Group/Pipeline name for which monitoring is required (For example. OK_US_Data_Load). Click Show Execution Logs.

    4. Spring cloud data flow page opens in a new tab. Click Jobs.

Troubleshooting

  • Mapping failure issues has to be fixed in the staging to canonical configurations. Make sure that Reltio attribute and data type is matching with it.

  • In case of failure in any task, fix the error and restart the task from the failed task till the end. If unable to identify or fix the error, contact MDM support team.

See DID of DDD data load to Reltio in Data Interface Documents.